RAPT

One of the most interesting, and least understood, fields of behavioral science involves the social substructure of daily life: friendship, politeness, impoliteness, relationship formation, and rapport. We all know that feeling of “getting along” or “clicking” with someone, and how that rapport builds as a relationship deepens, but there are few comprehensive theories about what that rapport-building process is and the mechanisms by which it takes place.

And yet, it has been shown that rapport plays an important role in everyday life: people learn more from teachers and peers with whom they feel rapport, have improved therapeutic benefits from therapists with whom they feel rapport, are more honest and more likely to complete surveys when they feel rapport with the interviewer, and so on. As virtual agents and AI take on an increasingly important and ubiquitous role in our lives, we believe that these systems should know how to build rapport so as to better support their users over time.

We know from research in the Learning Sciences that intelligent tutoring systems can help students learn, often vastly improving on learning in traditional, lecture-based classrooms [Kulik and Fletcher, 2016]. Furthermore, we know that when classroom peers collaborate on a learning project, those students who are friends collaborate more effectively than those students who are not friends [Azmitia and Montgomery, 1992].

When we combine these observations together, we conclude that a virtual peer tutor that can develop rapport with a student – or even engage in reciprocal peer tutoring (i.e. teaching and being taught by them) – may be of great use in supporting learning. Imagine a student who has rapport with their virtual tutor who takes more risks in learning, who asks for deeper explanations when they don’t understand, and who is more motivated to continue learning with their new virtual peer when they are frustrated. This is the system we are working towards.

In order to train our “machine-learning” artificial intelligence components of our virtual agent, we collect data from pairs of students tutoring each other. We collected data from 22 dyads of middle and early high school students tutoring each other in linear algebra over two weekly hour-long sessions. We had the students teach each other over Skype, so that the front-facing cameras would be better able to capture the students’ facial expressions, to aid in automatic detection and extraction of the nonverbal behaviors that contribute to and signal higher rapport, using the OpenFace toolkit for nonverbal behavior analysis.

We then used a combination of human annotation and machine learning natural language classifiers to identify the tutoring and learning strategies used by each student, to understand how students’ use of those strategies change as the rapport develops between the students.

Finally, we obtained high-frequency ratings of the rapport level between participants every thirty seconds, using third-party observers in a crowd-sourcing approach (Sinha and Cassell, 2015).

Crowdsourced Thin-Slice Rapport Rating Process

See here for more detail on previous studies of peer tutoring from the Articulab.

Having proposed a theoretical framework for rapport management, we have also developed a computational architecture that allows virtual agents to enhance, maintain, or even destroy long-term rapport with their users (Papangelis et al.,2014), and have implemented that architecture for a virtual tutoring agent in the RAPT project, as well as in a “Socially-Aware Robot Assistant”, in the SARA project (Matsuyama et al., 2016).

System Architecture for Rapport-Aware Virtual Tutor

Specifically, the RAPT system architecture contains the following modules:

1. The computational model of rapport: The computational model is the first to explain how humans in dyadic interactions build, maintain, and destroy rapport through the use of specific conversational strategies that function to fulfill specific social goals, and that are instantiated in particular verbal and nonverbal behaviors (Zhao et al., 2014).
2. Conversational strategy classification: The conversational strategy classifier can recognize high-level language strategies closely associated with social goals through training on linguistic and acoustic features associated with those conversational strategies in a test set (Zhao, Sinha, Black and Cassell ( SIGDIAL 2016)).
3. Rapport level estimation: The rapport estimator estimates the current rapport level between the user and the agent using temporal association rules (Zhao, Sinha, Black and Cassell (IVA 2016 Best Student Paper); Madaio, Lasko, Ogan, Cassell (EDM 2017)).
4. Social reasoning: The social reasoner outputs a social conversational strategy that the system must adopt in the current turn. The reasoner is modeled as a spreading activation network (Romero, Zhao, Matsuyama, Cassell (IJCAI 2017) ).
5. Natural language and nonverbal behavior generation: The natural language generation module expresses conversational strategies in specific language and associated nonverbal behaviors, and they are performed by a virtual agent (Cassell, et al., 2001).

Social Conversational Strategy Classifier

We have implemented a conversational strategy classifier to automatically recognize the user’s social conversational strategies – particular ways of talking, that contribute to building, maintaining or sometimes destroying a budding relationship. These include self-disclosure (SD), questions to elicit self-disclosure (QE), reference to shared experience (RSD), praise (PR), violation of social norms (VSN), and back-channel (BC). By including rich contextual features drawn from verbal, visual and vocal modalities of the speaker and interlocutor in the current and previous turns, we can successfully recognize these dialogue phenomena with an accuracy of over 80% and with a kappa of over 60%.

Rapport Estimator

We use the framework of temporal association rule learning to perform a fine-grained investigation into how sequences of interlocutor behaviors signal high and low interpersonal rapport. The behaviors analyzed include visual behaviors such as eye gaze and smiles, and both social and tutoring-related verbal conversational strategies, such as self-disclosure, social norm violation, praise, feedback, hints, instructions, and metacognitive reflection. We developed a forecasting model involving two-step fusion of learned temporal associated rules.

The estimation of rapport comprises two steps: in the first step, the approach is to learn the weighted contribution (vote) of each temporal association rule in predicting the presence/absence of a certain rapport state (via seven random-forest classifiers); in the second step, the intuition is to learn the weight corresponding to each of the binary classifiers for the rapport states, in order to predict the level of rapport (via multi-class classification model, or “Support Vector Machine”). Ground truth for the rapport state was obtained by having naive annotators rate the rapport between two interactants in the teen peer-tutoring corpus for every 30 second slice of an hour long interaction (those slices were randomized in order before being presented to the annotators so that ratings were of rapport states and not rapport deltas).

We are also currently experimenting with a different machine learning approach, using “temporally-selective attention models” in a neural network ensemble model. See Yu et al., (2017) for more details.

Temporally-Selective Attention Model for Rapport Estimation

Social Reasoner

The social reasoner is designed as a spreading activation model – a behavior network consisting of activation rules that govern which conversation strategy the system should adopt next. Taking as inputs the system’s intentions (e.g. “negative_feedback”, “provide_hint”), the history of the user’s conversational strategies, select non-verbal behaviours (e.g. head nod and smile) and the current rapport level, the activation energies will be updated, the Social Reasoner selects the system’s next conversational strategy, and sends both the system’s intent and conversational strategies to the NLG. The activation pre-conditions of the behavior network are extracted from the annotations carried out on the peer-tutoring corpus.

Verbal and Nonverbal Language Generation

Given the system’s intention (which includes the current conversational phase, the system intent, and the conversational strategy) these modules generate sentence and behavior plans. The Natural Language Generator (NLG) selects a certain syntactic template associated with the system’s intention from the sentence database. The templates are filled with content items. A generated sentence plan is sent to the BEAT (Behavior Expression Animation Toolkit), a nonverbal generation generator, and BEAT generates a behavior plan in the BML (Behavior Markup Language) form. The BML is then sent to SmartBody, which renders the required non-verbal behaviours on the Embodied Conversation Agent.

• Principal Investigators
  • Justine Cassell
  • Amy Ogan
  • Louis-Philippe Morency

• ArticuLab Members
  • Michael Madaio
  • Yoichi Matsuyama
  • Robert Huerbin
  • David Slebodnick
  • Tanmay Sinha
  • Ran Zhao

• Research Assistants: Michelina Astle, Jake Beley, Rishabh Chatterjee, Naomi Eigbe, Alvaro Granados, Rae Lasko, Tiffany Lee, Sarah Lehman, Jeffrey Li, William Liu, Kun Peng, Lynnette Ramsay, Anne Widom, Caroline Wu, Zian Zhao

RAPT (the Rapport-Aware Peer Tutor) is a collaboration among Professor Justine Cassell and her research group at Carnegie Mellon University, Amy Ogan, and Louis-Philippe Morency.

The research reported here was supported, in whole or in part, by the National Science Foundation Cyberlearning Award No. 1523162, and the Institute of Education Sciences, U.S. Department of Education, through Grant R305B150008 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.

Computing Power provided in part by DroneData